目標：從 iceberg 找到 spark 相關類就算成功
取得 plan：ReplaceData、MergeInto、DynamicFileFilterExec、ExtendedBatchScan
版本：spark 3.0.1，iceberg 0.11.0
數據源路徑：file:///Users/bjhl/tmp/icebergData
創建一個 maven 項目，pom.xml 文件如下<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.example</groupId> <artifactId>spark-3.x-worker</artifactId> <version>1.0-SNAPSHOT</version> <inceptionYear>2008</inceptionYear> <properties> <scala.version>2.12.8</scala.version> </properties>
<repositories> <repository> <id>scala-tools.org</id> <name>Scala-Tools Maven2 Repository</name> <url>http://scala-tools.org/repo-releases</url> </repository> </repositories>
<pluginRepositories> <pluginRepository> <id>scala-tools.org</id> <name>Scala-Tools Maven2 Repository</name> <url>http://scala-tools.org/repo-releases</url> </pluginRepository> </pluginRepositories>
<dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>${scala.version}</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.4</version> <scope>test</scope> </dependency> <dependency> <groupId>org.specs</groupId> <artifactId>specs</artifactId> <version>1.2.5</version> <scope>test</scope> </dependency>
<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>3.0.1</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.12</artifactId> <version>3.0.1</version> <scope>provided</scope> </dependency>
<dependency> <groupId>org.apache.iceberg</groupId> <artifactId>iceberg-spark3-runtime</artifactId> <version>0.11.0</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>1.9.2</version> </dependency> </dependencies>
<build> <sourceDirectory>src/main/scala</sourceDirectory> <testSourceDirectory>src/test/scala</testSourceDirectory> <plugins> <plugin> <groupId>org.scala-tools</groupId> <artifactId>maven-scala-plugin</artifactId> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> </execution> </executions> <configuration> <scalaVersion>${scala.version}</scalaVersion> <args> <arg>-target:jvm-1.5</arg> </args> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-eclipse-plugin</artifactId> <configuration> <downloadSources>true</downloadSources> <buildcommands> <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand> </buildcommands> <additionalProjectnatures> <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature> </additionalProjectnatures> <classpathContainers> <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer> <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer> </classpathContainers> </configuration> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>6</source> <target>6</target> </configuration> </plugin> </plugins> </build> <reporting> <plugins> <plugin> <groupId>org.scala-tools</groupId> <artifactId>maven-scala-plugin</artifactId> <configuration> <scalaVersion>${scala.version}</scalaVersion> </configuration> </plugin> </plugins> </reporting></project>SparkSession 配置val spark = SparkSession .builder() .config("spark.sql.catalog.hadoop_prod.type", "hadoop") .config("spark.sql.catalog.hadoop_prod", "org.apache.iceberg.spark.SparkSessionCatalog") .config("spark.sql.catalog.hadoop_prod.warehouse", "file:///Users/bjhl/tmp/icebergData") .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") .appName(this.getClass.getSimpleName) .master("local[*]") .getOrCreate()創建一張表val hdpCatalog = spark.sessionState.catalogManager.catalog("hadoop_prod").asInstanceOf[SparkCatalog] val namespaces = Array("test") val identifier = new SimpleLocalIdentifierImpl("/Users/bjhl/tmp/icebergData/test/table_a", namespaces) val options = new util.HashMap[String, String]() val schema = new StructType() .add("c1", IntegerType, true) .add("c2", StringType, true) .add("c3", StringType, true) hdpCatalog.createTable(identifier, schema, null, options) spark.sql("insert into hadoop_prod.test.table_a VALUES (1, \"wlq\",\"zyc\")")生成的結構如下包含元數據信息和數據信息，test 類比 庫名，table_a 是表名
讀取並更新，列印執行計劃// 獲取 表結構信息 val df = spark.table("hadoop_prod.test.table_a") df.printSchema()
df.show()// val dfTableA = spark.read.format("iceberg").load("/Users/bjhl/tmp/icebergData/default/table_a")// dfTableA.show()
spark.sql("merge into hadoop_prod.test.table_a t " + "using (select 1 as c1, \"zyc\" as c2, \"wlq\" as c3) s " + "on t.c1 = s.c1 " + "when matched " + "then update set t.c3 = s.c3").explain()
df.show()
println("讀寫取 iceberg 數據結束")注意：這裡直接 read.format 方式一直使用的是 HiveCatalog 去獲取信息，老是報錯，目前還沒定位出問題
效果如下：
更新數據後，存儲路徑目錄變化如下
元數據和數據都有新增相應的版本，猜測是以快照的方式實現？
表結構
更新前數據
更新後數據
重點：物理執行計劃，如下
結合 icebergclone iceberg 代碼構建下，上面的類來自 iceberg-spark3-extensions
後面就是根據代碼驗證猜想的過程
結束語注意：
"spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" 要設置，才能支持 merge into 等功能
疑問：
1. iceberg 是以什麼方式做的更新？
2. 對於 iceberg 的存儲方式，spark 任務的運行過程哪個階段性能有所提升或者有所下降？
3. 對於 iceberg 的實現方式，spark 基於其做了哪些優化？