A table is registered with the following code:

Both users and orders are Delta Lake tables. Which statement describes the results of querying recent_orders?

• A. The versions of each source table will be stored in the table transaction log; query results will be saved to DBFS with each query.
• B. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query finishes.
• C. Results will be computed and cached when the table is defined; these cached results will incrementally update as new records are inserted into source tables.
• D. All logic will execute when the table is defined and store the result of joining tables to the DBFS; this stored data will be returned when the table is queried.
• E. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query began.

A data engineer is reviewing the PySpark code to copy a part of the production dataset to the sandbox environment, and needs to be sure that no PII(Personally Identifiable Information) data is being copied. After checking the sales table, the data engineer notices that it has user emails as the only PII data included as well as being the only column to identify the user.
from pyspark.sql import functions as F

Which anonymised code should be used to achieve the required outcome?

• A. df.withColumn ("user_emai", F.expr("uuid()"))
• B. df.withColumn ("user_email", F.sha2 ("user_email"))
• C. df.withColumn ("user_email", F.regexp_replace ("user_eamail", "@*", "@anonymized.com"))
• D. df.withColumn ("hashed_email", sha2 ("user_email"))

When monitoring a complex workload, being able to see the query plan is critical to understanding what the workload is doing. Where can the visualization of the query plan be found?

• A. In the Spark UI, under the SQL/DataFrame tab
• B. In the Query Profiler, under Query Source
• C. In the Spart UI, under the Jobs tab
• D. In the Query Profiler, under the Stages tab

A data engineer is working in an interactive notebook with many transformations before outputting the result from display(df.collect() ). The notebook includes wide transformations and a cross join.
The data engineer is getting the following error: "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached." Which action should the data engineer take?

• A. Run the notebook on a single node cluster to keep driver from falling.
• B. Check into the Spark UI to see how many jobs are assigned to each stage as they are employing fewer executors.
• C. Rewrite their code to avoid putting memory pressure on the driver node.
• D. Look at the compute metrics UI to see if the executors have higher than 90% memory utilization.

An hourly batch job is configured to ingest data files from a cloud object storage container where each batch represent all records produced by the source system in a given hour. The batch job to process these records into the Lakehouse is sufficiently delayed to ensure no late-arriving data is missed. The user_id field represents a unique key for the data, which has the following schema:
user_id BIGINT, username STRING, user_utc STRING, user_region STRING, last_login BIGINT, auto_pay BOOLEAN, last_updated BIGINT New records are all ingested into a table named account_history which maintains a full record of all data in the same schema as the source. The next table in the system is named account_current and is implemented as a Type 1 table representing the most recent value for each unique user_id.
Assuming there are millions of user accounts and tens of thousands of records processed hourly, which implementation can be used to efficiently update the described account_current table as part of each hourly batch job?

• A. Filter records in account history using the last updated field and the most recent hour processed, making sure to deduplicate on username; write a merge statement to update or insert the most recent value for each username.
• B. Use Auto Loader to subscribe to new files in the account history directory; configure a Structured Streaminq trigger once job to batch update newly detected files into the account current table.
• C. Filter records in account history using the last updated field and the most recent hour processed, as well as the max last iogin by user id write a merge statement to update or insert the most recent value for each user id.
• D. Overwrite the account current table with each batch using the results of a query against the account history table grouping by user id and filtering for the max value of last updated.
• E. Use Delta Lake version history to get the difference between the latest version of account history and one version prior, then write these records to account current.