Databricks Certified Associate Developer for Apache Spark 3.0 認定 Associate-Developer-Apache-Spark 試験問題:

1. Which of the following describes properties of a shuffle?

A) Shuffles belong to a class known as "full transformations".
B) A shuffle is one of many actions in Spark.
C) Shuffles involve only single partitions.
D) In a shuffle, Spark writes data to disk.
E) Operations involving shuffles are never evaluated lazily.

2. Which of the following code blocks returns a DataFrame that is an inner join of DataFrame itemsDf and DataFrame transactionsDf, on columns itemId and productId, respectively and in which every itemId just appears once?

A) itemsDf.join(transactionsDf, "itemsDf.itemId==transactionsDf.productId").distinct("itemId")
B) itemsDf.join(transactionsDf, "itemsDf.itemId==transactionsDf.productId", how="inner").dropDuplicates(["itemId"])
C) itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId).dropDuplicates(["itemId"])
D) itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId).dropDuplicates("itemId")
E) itemsDf.join(transactionsDf, itemsDf.itemId==transactionsDf.productId, how="inner").distinct(["itemId"])

3. The code block displayed below contains an error. The code block should save DataFrame transactionsDf at path path as a parquet file, appending to any existing parquet file. Find the error.
Code block:

A) save() is evaluated lazily and needs to be followed by an action.
B) The mode option should be omitted so that the command uses the default mode.
C) The code block is missing a reference to the DataFrameWriter.
D) transactionsDf.format("parquet").option("mode", "append").save(path)
E) The code block is missing a bucketBy command that takes care of partitions.
F) Given that the DataFrame should be saved as parquet file, path is being passed to the wrong method.

4. Which of the following code blocks creates a new DataFrame with two columns season and wind_speed_ms where column season is of data type string and column wind_speed_ms is of data type double?

A) spark.createDataFrame({"season": ["winter","summer"], "wind_speed_ms": [4.5, 7.5]})
B) spark.createDataFrame([("summer", 4.5), ("winter", 7.5)], ["season", "wind_speed_ms"])
C) spark.newDataFrame([("summer", 4.5), ("winter", 7.5)], ["season", "wind_speed_ms"])
D) 1. from pyspark.sql import types as T
2. spark.createDataFrame((("summer", 4.5), ("winter", 7.5)), T.StructType([T.StructField("season",
E) spark.DataFrame({"season": ["winter","summer"], "wind_speed_ms": [4.5, 7.5]})
F) CharType()), T.StructField("season", T.DoubleType())]))

5. Which of the following code blocks returns approximately 1000 rows, some of them potentially being duplicates, from the 2000-row DataFrame transactionsDf that only has unique rows?

A) transactionsDf.sample(False, 0.5)
B) transactionsDf.sample(True, 0.5, force=True)
C) transactionsDf.take(1000).distinct()
D) transactionsDf.sample(True, 0.5)
E) transactionsDf.take(1000)

質問と回答：