Databricks data-engineer-professional

Choose your answer

• A

  Decrease the trigger interval to 5 seconds; triggering batches more frequently allows idle executors to begin processing the next batch while longer running tasks from previous batches finish.

• B

  Increase the trigger interval to 30 seconds; setting the trigger interval near the maximum execution time observed for each batch is always best practice to ensure no records are dropped.

• C

  The trigger interval cannot be modified without modifying the checkpoint directory; to maintain the current stream state, increase the number of shuffle partitions to maximize parallelism.

• D

  Use the trigger once option and configure a Databricks job to execute the query every 10 seconds; this ensures all backlogged records are processed with each batch.

• E

  Decrease the trigger interval to 5 seconds; triggering batches more frequently may prevent records from backing up and large batches from causing spill.

Pick an answer to check it, or press Show answer to see the correct one.

Choose your answer

• A

  An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an OPTIMIZE job is executed toward a default of 1 GB.

• B

  Before a Jobs cluster terminates, OPTIMIZE is executed on all tables modified during the most recent job.

• C

  Optimized writes use logical partitions instead of directory partitions; because partition boundaries are only represented in metadata, fewer small files are written.

• D

  Data is queued in a messaging bus instead of committing data directly to memory; all data is committed from the messaging bus in one batch once the job is complete.

• E

  An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an OPTIMIZE job is executed toward a default of 128 MB.

Pick an answer to check it, or press Show answer to see the correct one.

Choose your answer

• A

  Structured Streaming leverages the parallel processing of GPUs to achieve highly parallel data throughput.

• B

  Structured Streaming is implemented as a messaging bus and is derived from Apache Kafka.

• C

  Structured Streaming uses specialized hardware and I/O streams to achieve sub-second latency for data transfer.

• D

  Structured Streaming models new data arriving in a data stream as new rows appended to an unbounded table.

• E

  Structured Streaming relies on a distributed network of nodes that hold incremental state values for cached stages.

Pick an answer to check it, or press Show answer to see the correct one.

Choose your answer

• A

  spark.sql.files.maxPartitionBytes

• B

  spark.sql.autoBroadcastJoinThreshold

• C

  spark.sql.files.openCostInBytes

• D

  spark.sql.adaptive.coalescePartitions.minPartitionNum

• E

  spark.sql.adaptive.advisoryPartitionSizeInBytes

Pick an answer to check it, or press Show answer to see the correct one.

Choose your answer

• A

  Task queueing resulting from improper thread pool assignment.

• B

  Spill resulting from attached volume storage being too small.

• C

  Network latency due to some cluster nodes being in different regions from the source data

• D

  Skew caused by more data being assigned to a subset of spark-partitions.

• E

  Credential validation errors while pulling data from an external system.

Pick an answer to check it, or press Show answer to see the correct one.