既然你想和去年比较 year 应为按列排序。但也要保持 movie_name 以及 language ，它们应该是按列划分的。
将该行的评级与最近一年的评级进行比较 lag 函数用于 date 正在上升。

import pyspark.sql.functions as f
from pyspark.sql import Window

w = Window.partitionBy('movie_name', 'language').orderBy('year')

df.withColumn('lead', f.lag('rating', 1).over(w)) \
  .withColumn('rating_diff', f.when(f.col('lead').isNotNull(), f.col('rating') - f.col('lead')).otherwise(f.lit(None))) \
  .show(10, False)

+----+----------+--------+------+----+-----------+
|year|movie_name|language|rating|lead|rating_diff|
+----+----------+--------+------+----+-----------+
|2019|antony    |kannada |9     |null|null       |
|2020|antony    |kannada |8     |9   |-1         |
|2019|akash     |english |10    |null|null       |
|2020|akash     |english |10    |10  |0          |
+----+----------+--------+------+----+-----------+