最新のAssociate-Developer-Apache-Spark-3.5問題集で試験を準備する

A data engineer needs to persist a file-based data source to a specific location. However, by default, Spark writes to the warehouse directory (e.g., /user/hive/warehouse). To override this, the engineer must explicitly define the file path.
Which line of code ensures the data is saved to a specific location?
Options:

• A. users.write(path="/some/path").saveAsTable("default_table")
• B. users.write.saveAsTable("default_table", path="/some/path")
• C. users.write.option("path", "/some/path").saveAsTable("default_table")
• D. users.write.saveAsTable("default_table").option("path", "/some/path")

48 of 55.
A data engineer needs to join multiple DataFrames and has written the following code:
from pyspark.sql.functions import broadcast
data1 = [(1, "A"), (2, "B")]
data2 = [(1, "X"), (2, "Y")]
data3 = [(1, "M"), (2, "N")]
df1 = spark.createDataFrame(data1, ["id", "val1"])
df2 = spark.createDataFrame(data2, ["id", "val2"])
df3 = spark.createDataFrame(data3, ["id", "val3"])
df_joined = df1.join(broadcast(df2), "id", "inner") \
.join(broadcast(df3), "id", "inner")
What will be the output of this code?

• A. The code will result in an error because broadcast() must be called before the joins, not inline.
• B. The code will fail because only one broadcast join can be performed at a time.
• C. The code will work correctly and perform two broadcast joins simultaneously to join df1 with df2, and then the result with df3.
• D. The code will fail because the second join condition (df2.id == df3.id) is incorrect.

A developer needs to produce a Python dictionary using data stored in a small Parquet table, which looks like this:

The resulting Python dictionary must contain a mapping of region -> region id containing the smallest 3 region_id values.
Which code fragment meets the requirements?
A)

B)

C)

D)

The resulting Python dictionary must contain a mapping of region -> region_id for the smallest 3 region_id values.
Which code fragment meets the requirements?

• A. regions = dict(
  regions_df
  .select('region_id', 'region')
  .limit(3)
  .collect()
  )
• B. regions = dict(
  regions_df
  .select('region', 'region_id')
  .sort(desc('region_id'))
  .take(3)
  )
• C. regions = dict(
  regions_df
  .select('region', 'region_id')
  .sort('region_id')
  .take(3)
  )
• D. regions = dict(
  regions_df
  .select('region_id', 'region')
  .sort('region_id')
  .take(3)
  )

A Spark engineer must select an appropriate deployment mode for the Spark jobs.
What is the benefit of using cluster mode in Apache Spark™?

• A. In cluster mode, the driver program runs on one of the worker nodes, allowing the application to fully utilize the distributed resources of the cluster.
• B. In cluster mode, resources are allocated from a resource manager on the cluster, enabling better performance and scalability for large jobs
• C. In cluster mode, the driver is responsible for executing all tasks locally without distributing them across the worker nodes.
• D. In cluster mode, the driver runs on the client machine, which can limit the application's ability to handle large datasets efficiently.

2 of 55. Which command overwrites an existing JSON file when writing a DataFrame?

• A. df.write.mode("overwrite").json("path/to/file")
• B. df.write.mode("append").json("path/to/file")
• C. df.write.option("overwrite").json("path/to/file")
• D. df.write.json("path/to/file")