/venv/lib/python3.10/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
pyspark.sql.utils.AnalysisException: The column number of the existing table analytics_platform_eng.image_suggestions_suggestions(struct<page_id:bigint,id:string,image:string,origin_wiki:string,confidence:int,found_on:array<string>,kind:array<string>,page_rev:bigint,snapshot:string,wiki:string>) doesn't match the data schema(struct<page_id:bigint,id:string,image:string,origin_wiki:string,confidence:int,found_on:array<string>,kind:array<string>,page_rev:bigint,section_heading:string,snapshot:string,wiki:string>)

 hostname -f
an-airflow1004.eqiad.wmnet

sudo -u analytics-platform-eng spark3-sql

spark-sql (default)> use analytics_platform_eng;
Response code
Time taken: 2.896 seconds

spark-sql (default)> ALTER TABLE `image_suggestions_suggestions` ADD COLUMNS (
                   >     `section_heading` string
                   > );
Response code
Time taken: 0.795 seconds

spark-sql (default)> describe formatted image_suggestions_suggestions;
col_name	data_type	comment
page_id	bigint	Uniquely identifying primary key along with wiki. This value is preserved across edits, renames, and, as of MediaWiki 1.27, deletions, via an analogous field in the archive table (introduced in MediaWiki 1.11). For example, for this page, page_id = 10501. 
id	string	NULL
image	string	The sanitized page title, without the namespace, with a maximum of 255 characters (binary). It is stored as text, with spaces replaced by underscores. The real title shown in articles is just this title with underscores (_) converted to spaces ( ). For exa
origin_wiki	string	NULL
confidence	int	NULL
found_on	array<string>	NULL
kind	array<string>	NULL
page_rev	bigint	NULL
section_heading	string	NULL
snapshot	string	NULL
wiki	string	The wiki_db project
# Partition Information		
# col_name	data_type	comment
snapshot	string	NULL
wiki	string	The wiki_db project
		
# Detailed Table Information		
Database	analytics_platform_eng	
Table	image_suggestions_suggestions	
Owner	analytics-platform-eng	
Created Time	Fri May 20 13:21:14 UTC 2022	
Last Access	UNKNOWN	
Created By	Spark 2.4.4	
Type	MANAGED	
Provider	parquet	
Location	hdfs://analytics-hadoop/user/hive/warehouse/analytics_platform_eng.db/image_suggestions_suggestions	
Serde Library	org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe	
InputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat	
OutputFormat	org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat	
Partition Provider	Catalog	
Time taken: 0.17 seconds, Fetched 30 row(s)