NameDefaultMeaning spark.executor.instances 1 The number of executors for static allocation spark.executor.cores 1 The number of cores to use on each executor spark.yarn.am.memory 512m Amount of memory to use for the YARN Application Master in client mode spark.yarn.am.memoryOver...
The sql query has already finished since one join side is empty (Spark AQE optimzie it). The running job/stage is the other join side shuffle stage which has no meaning. BTW, if your Spark version is less than 3.4, the Spark sql execution status may be wrong, see SPARK-40834. 👍 ...
If not, it will init a HiveClient and create bunches of Hive objects. I did overlook this point and executedkyuubi.engine.initialize.sql=select 1before the statistics. The screenshot below shows the state after I unsetkyuubi.engine.initialize.sql, meaning no SQL was executed. The overall diff...