在文章tensorflow(5)將ckpt轉化為pb文件並利用tensorflow/serving實現模型部署及預測中，筆者以一個簡單的例子，來介紹如何在tensorflow中將ckpt轉化為pb文件，並利用tensorflow/serving來實現模型部署及預測。本文將會介紹如何使用tensorflow/serving來實現單模型部署、多模型部署、模型版本控制以及模型預測。
我們將會以Docker形式使用tensorflow/serving，因此需要在你的環境中安裝好Docker。我們以tensorflow/serving:1.14.0為例，所以我們需要拉取這個鏡像到本地：
docker pull tensorflow/serving:1.14.0
本文演示的項目結構如下：
項目結構單模型部署及預測 創建模型我們先使用Tensorflow創建模型：z=x*y+t，其中x,y等於2.0，t為變量，z為輸出結果。創建模型腳本（single_model.py）的完整代碼如下：
# -*- coding: utf-8 -*-
import tensorflow as tf
g = tf.Graph()
with g.as_default() as g:
x = tf.Variable(2.0, dtype=tf.float32, name="x")
y = tf.Variable(2.0, dtype=tf.float32, name="y")
xy = x * y
t = tf.placeholder(shape=None, dtype=tf.float32, name="t")
z = tf.add(xy, t, name="z")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
result = sess.run(z, feed_dict={t: 1.0})
print("result: ", result)
# save the model
saver = tf.train.Saver()
saver.save(sess, save_path='./ckpt_models/add/add.ckpt')
輸出結果為：
result: 5.0
上述模型生成pb文件的完整代碼（single_ckpt_2_pb.py）如下：
# -*- coding: utf-8 -*-
import tensorflow as tf
from tensorflow.python import saved_model
export_path = "pb_models/add/1"
graph = tf.Graph()
saver = tf.train.import_meta_graph("./ckpt_models/add/add.ckpt.meta", graph=graph)
with tf.Session(graph=graph) as sess:
saver.restore(sess, tf.train.latest_checkpoint("./ckpt_models/add"))
saved_model.simple_save(session=sess,
export_dir=export_path,
inputs={"t": graph.get_operation_by_name('t').outputs[0]},
outputs={"z": graph.get_operation_by_name('z').outputs[0]})
運行上述代碼，會在pb_models文件夾下生成文件如下：
$ tree pb_models/
pb_models/
├── add
│ └── 1
│ ├── saved_model.pb
│ └── variables
│ ├── variables.data-00000-of-00001
│ └── variables.index
接著我們使用Docker部署該模型，命令如下：
docker run -t --rm -p 8551:8501 -v "absolute_path_to_pb_models/pb_models/add:/models/add" -e MODEL_NAME=add tensorflow/serving:1.14.0
其中absolute_path_to_pb_models為pb_models所在的完整路徑。輸出結果如下：
2021-01-06 04:14:32.790794: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config: model_name: add model_base_path: /models/add
2021-01-06 04:14:32.810015: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2021-01-06 04:14:32.810054: I tensorflow_serving/model_servers/server_core.cc:561] (Re-)adding model: add
2021-01-06 04:14:32.910293: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: add version: 1}
2021-01-06 04:14:32.910317: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: add version: 1}
2021-01-06 04:14:32.910327: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: add version: 1}
2021-01-06 04:14:32.910373: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: /models/add/1
2021-01-06 04:14:32.910385: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /models/add/1
2021-01-06 04:14:32.910573: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2021-01-06 04:14:32.916524: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-01-06 04:14:32.956639: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2021-01-06 04:14:33.075419: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 165028 microseconds.
2021-01-06 04:14:33.075456: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:103] No warmup data file found at /models/add/1/assets.extra/tf_serving_warmup_requests
2021-01-06 04:14:33.075528: I tensorflow_serving/core/loader_harness.cc:86] Successfully loaded servable version {name: add version: 1}
2021-01-06 04:14:33.084877: I tensorflow_serving/model_servers/server.cc:324] Running gRPC ModelServer at 0.0.0.0:8500 ...
[warn] getaddrinfo: address family for nodename not supported
2021-01-06 04:14:33.117161: I tensorflow_serving/model_servers/server.cc:344] Exporting HTTP/REST API at:localhost:8501 ...
[evhttp_server.cc : 239] RAW: Entering the event loop ...
出現上述結果即表示模型部署成功，其中8500為gRPC埠，8501為HTTP埠，這裡我們只使用了HTTP埠。
模型預測查看模型部署狀態的curl命令：
curl http://192.168.1.193:8551/v1/models/add
輸出結果為：
{
"model_version_status": [
{
"version": "1",
"state": "AVAILABLE",
"status": {
"error_code": "OK",
"error_message": ""
}
}
]
}
查看模型部署元數據的curl命令：
curl http://192.168.1.193:8551/v1/models/add/metadata
輸出結果為：
{
"model_spec": {
"name": "add",
"signature_name": "",
"version": "1"
},
"metadata": {
"signature_def": {
"signature_def": {
"serving_default": {
"inputs": {
"t": {
"dtype": "DT_FLOAT",
"tensor_shape": {
"dim": [],
"unknown_rank": true
},
"name": "t:0"
}
},
"outputs": {
"z": {
"dtype": "DT_FLOAT",
"tensor_shape": {
"dim": [],
"unknown_rank": true
},
"name": "z:0"
}
},
"method_name": "tensorflow/serving/predict"
}
}
}
}
}
從中我們可以看到模型的輸入與輸出。
使用curl命令進行模型預測：
curl --location --request POST 'http://192.168.1.193:8551/v1/models/add:predict' \
--header 'Content-Type: application/json' \
--data-raw '{
"instances": [{"t": 2.0}]
}'
輸出結果為：
{
"predictions": [
6.0
]
}
如果使用Python進行模型預測，其代碼（single_tf_serving.py）如下：
# -*- coding: utf-8 -*-
import requests
# 利用tensorflow/serving的HTTP接口請求進行預測
t = 2.0
tensor = {"instances": [{"t": t}]}
url = "http://192.168.1.193:8551/v1/models/add:predict"
req = requests.post(url, json=tensor)
if req.status_code == 200:
z = req.json()['predictions'][0]
print("model_add:", z)
我們先使用Tensorflow創建多個模型：z=x*y+t,z=x*y-t,z=x*y*t,z=x*y/t,，其中x,y等於2.0，t為變量，z為輸出結果。創建模型腳本（multi_model.py）的完整代碼如下：
# -*- coding: utf-8 -*-
import tensorflow as tf
# add model
with tf.Graph().as_default() as g:
x = tf.Variable(2.0, dtype=tf.float32, name="x")
y = tf.Variable(2.0, dtype=tf.float32, name="y")
xy = x * y
t = tf.placeholder(shape=None, dtype=tf.float32, name="t")
z = tf.add(xy, t, name="z")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
result = sess.run(z, feed_dict={t: 3.0})
print("result: ", result)
# save the model
saver = tf.train.Saver()
saver.save(sess, save_path='./ckpt_models/add/add.ckpt')
# substract model
with tf.Graph().as_default() as g:
x = tf.Variable(2.0, dtype=tf.float32, name="x")
y = tf.Variable(2.0, dtype=tf.float32, name="y")
xy = x * y
t = tf.placeholder(shape=None, dtype=tf.float32, name="t")
z = tf.subtract(xy, t, name="z")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
result = sess.run(z, feed_dict={t: 3.0})
print("result: ", result)
# save the model
saver = tf.train.Saver()
saver.save(sess, save_path='./ckpt_models/subtract/subtract.ckpt')
# multipy
with tf.Graph().as_default() as g:
x = tf.Variable(2.0, dtype=tf.float32, name="x")
y = tf.Variable(2.0, dtype=tf.float32, name="y")
xy = x * y
t = tf.placeholder(shape=None, dtype=tf.float32, name="t")
z = tf.multiply(xy, t, name="z")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
result = sess.run(z, feed_dict={t: 3.0})
print("result: ", result)
# save the model
saver = tf.train.Saver()
saver.save(sess, save_path='./ckpt_models/multiply/multiply.ckpt')
# divide model
with tf.Graph().as_default() as g:
x = tf.Variable(2.0, dtype=tf.float32, name="x")
y = tf.Variable(2.0, dtype=tf.float32, name="y")
xy = x * y
t = tf.placeholder(shape=None, dtype=tf.float32, name="t")
z = tf.divide(xy, t, name="z")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
result = sess.run(z, feed_dict={t: 3.0})
print("result: ", result)
# save the model
saver = tf.train.Saver()
saver.save(sess, save_path='./ckpt_models/divide/divide.ckpt')
輸出結果如下：
result: 7.0
result: 1.0
result: 12.0
result: 1.3333334
上述模型生成pb文件的完整代碼（multi_ckpt_2_pb.py）如下：（注意，運行代碼前需刪除pb_models下的add目錄，不然會報錯。）
# -*- coding: utf-8 -*-
import tensorflow as tf
from tensorflow.python import saved_model
# change ckpt file to pb file
def model_export(model_name):
export_path = "pb_models/{}/1".format(model_name)
graph = tf.Graph()
saver = tf.train.import_meta_graph("./ckpt_models/{}/{}.ckpt.meta".format(model_name, model_name),
graph=graph)
with tf.Session(graph=graph) as sess:
saver.restore(sess, tf.train.latest_checkpoint("./ckpt_models/{}".format(model_name)))
saved_model.simple_save(session=sess,
export_dir=export_path,
inputs={"t": graph.get_operation_by_name('t').outputs[0]},
outputs={"z": graph.get_operation_by_name('z').outputs[0]})
model_export("add")
model_export("subtract")
model_export("multiply")
model_export("divide")
運行上述代碼，會在pb_models文件夾下生成文件如下：
$ tree pb_models/
pb_models/
├── add
│ └── 1
│ ├── saved_model.pb
│ └── variables
│ ├── variables.data-00000-of-00001
│ └── variables.index
├── divide
│ └── 1
│ ├── saved_model.pb
│ └── variables
│ ├── variables.data-00000-of-00001
│ └── variables.index
├── models.config
├── multiply
│ └── 1
│ ├── saved_model.pb
│ └── variables
│ ├── variables.data-00000-of-00001
│ └── variables.index
└── subtract
└── 1
├── saved_model.pb
└── variables
├── variables.data-00000-of-00001
└── variables.index
12 directories, 13 files
tensorflow/serving支持多個模型同時部署，其部署命令如下：
docker run -t -d --rm -p 8551:8501 -v "absolute_path_to_pb_models/pb_models:/models" tensorflow/serving:1.14.0 --model_config_file=/models/models.config
其中absolute_path_to_pb_models為pb_models所在的完整路徑。需要註明models.config文件，正是在這個文件中，我們配置了多個模型的部署信息，文件內容如下:
model_config_list {
config {
name: "add"
base_path: "/models/add"
model_platform: "tensorflow"
},
config {
name: "subtract"
base_path: "/models/subtract"
model_platform: "tensorflow"
},
config {
name: "multiply"
base_path: "/models/multiply"
model_platform: "tensorflow"
},
config {
name: "divide"
base_path: "/models/divide"
model_platform: "tensorflow"
}
}
使用Python腳本進行模型預測的代碼（multi_tf_serving.py）如下：
# -*- coding: utf-8 -*-
import tensorflow as tf
from tensorflow.python import saved_model
# change ckpt file to pb file
def model_export(model_name):
export_path = "pb_models/{}/1".format(model_name)
graph = tf.Graph()
saver = tf.train.import_meta_graph("./ckpt_models/{}/{}.ckpt.meta".format(model_name, model_name),
graph=graph)
with tf.Session(graph=graph) as sess:
saver.restore(sess, tf.train.latest_checkpoint("./ckpt_models/{}".format(model_name)))
saved_model.simple_save(session=sess,
export_dir=export_path,
inputs={"t": graph.get_operation_by_name('t').outputs[0]},
outputs={"z": graph.get_operation_by_name('z').outputs[0]})
model_export("add")
model_export("subtract")
model_export("multiply")
model_export("divide")
輸出結果如下：
model_add: 8.0
model_subtract: 0.0
model_multiply: 16.0
model_divide: 1.0
tensorflow/serving還支持同一個模型的不同版本的部署。我們在這裡創建一個add模型，其有三個版本，分別為：z=x*y+t,z=x*y+2t,z=x*y+3t,，其中x,y等於2.0，t為變量，z為輸出結果。創建模型腳本（version_control_model.py）的完整代碼如下：
# -*- coding: utf-8 -*-
import tensorflow as tf
# 第一個模型
g = tf.Graph()
with g.as_default() as g:
x = tf.Variable(2.0, dtype=tf.float32, name="x")
y = tf.Variable(2.0, dtype=tf.float32, name="y")
xy = x * y
t = tf.placeholder(shape=None, dtype=tf.float32, name="t")
z = tf.add(xy, t, name="z")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
result = sess.run(z, feed_dict={t: 1.0})
print("result: ", result)
# save the model
saver = tf.train.Saver()
saver.save(sess, save_path='./ckpt_models/add/add.ckpt')
# 第二個模型
g = tf.Graph()
with g.as_default() as g:
x = tf.Variable(2.0, dtype=tf.float32, name="x")
y = tf.Variable(2.0, dtype=tf.float32, name="y")
xy = x * y
t = tf.placeholder(shape=None, dtype=tf.float32, name="t")
z = tf.add(xy, 2*t, name="z")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
result = sess.run(z, feed_dict={t: 1.0})
print("result: ", result)
# save the model
saver = tf.train.Saver()
saver.save(sess, save_path='./ckpt_models/add/add2.ckpt')
# 第三個模型
g = tf.Graph()
with g.as_default() as g:
x = tf.Variable(2.0, dtype=tf.float32, name="x")
y = tf.Variable(2.0, dtype=tf.float32, name="y")
xy = x * y
t = tf.placeholder(shape=None, dtype=tf.float32, name="t")
z = tf.add(xy, 3*t, name="z")
with tf.Session(graph=g) as sess:
sess.run(tf.global_variables_initializer())
result = sess.run(z, feed_dict={t: 1.0})
print("result: ", result)
# save the model
saver = tf.train.Saver()
saver.save(sess, save_path='./ckpt_models/add/add3.ckpt')
輸出結果如下：
result: 5.0
result: 6.0
result: 7.0
上述模型生成pb文件的完整代碼（version_control_ckpt_2_pb.py）如下：（注意，運行代碼前需刪除pb_models下的add目錄，不然會報錯。）
# -*- coding: utf-8 -*-
import tensorflow as tf
from tensorflow.python import saved_model
# change ckpt file to pb file
def model_export(model_version, tf_version):
export_path = "pb_models/add/{}".format(tf_version)
graph = tf.Graph()
saver = tf.train.import_meta_graph("./ckpt_models/add/{}.ckpt.meta".format(model_version),
graph=graph)
with tf.Session(graph=graph) as sess:
saver.restore(sess, tf.train.latest_checkpoint("./ckpt_models/add"))
saved_model.simple_save(session=sess,
export_dir=export_path,
inputs={"t": graph.get_operation_by_name('t').outputs[0]},
outputs={"z": graph.get_operation_by_name('z').outputs[0]})
model_export("add", 1)
model_export("add2", 2)
model_export("add3", 3)
運行上述代碼，會在pb_models文件夾下生成文件如下：
pb_models/
├── add
│ ├── 1
│ │ ├── saved_model.pb
│ │ └── variables
│ │ ├── variables.data-00000-of-00001
│ │ └── variables.index
│ ├── 2
│ │ ├── saved_model.pb
│ │ └── variables
│ │ ├── variables.data-00000-of-00001
│ │ └── variables.index
│ └── 3
│ ├── saved_model.pb
│ └── variables
│ ├── variables.data-00000-of-00001
│ └── variables.index
└── models.config
7 directories, 10 files
tensorflow/serving支持模型的多個版本同時部署，其部署命令如下：
docker run -t -d --rm -p 8551:8501 -v "absolute_path_to_pb_models/pb_models:/models" tensorflow/serving:1.14.0 --model_config_file=/models/models.config
其中absolute_path_to_pb_models為pb_models所在的完整路徑。需要註明models.config文件，正是在這個文件中，我們配置了add模型的版本部署信息，文件內容如下:
model_config_list {
config {
name: "add"
base_path: "/models/add"
model_platform: "tensorflow"
model_version_policy{
all{
}
}
}
}
注意model_version_policy欄位，這裡選擇all，表示部署所有版本的模型。model_version_policy：模型版本策略，支持如下三種：
all：同時服務所有版本
latest：默認，默認值：1
specific：服務指定版本，可同時配置多個
模型預測使用Python腳本進行模型預測的代碼（version_control_tf_serving.py）如下：
# -*- coding: utf-8 -*-
import requests
# 利用tensorflow/serving的HTTP接口請求進行預測
def model_predict(model_version):
t = 4.0
tensor = {"instances": [{"t": t}]}
url = "http://192.168.1.193:8551/v1/models/add/versions/{}:predict".format(model_version)
req = requests.post(url, json=tensor)
if req.status_code == 200:
z = req.json()['predictions'][0]
print("model_version{}: ".format(model_version), z)
model_predict("1")
model_predict("2")
model_predict("3")
輸出結果如下：
model_version1: 8.0
model_version2: 12.0
model_version3: 16.0
注意不同版本的模型請求url是不同的，規則如下：
/v1/models/<model name>/versions/<version number>
本項目已經上傳至Github，網址為：https://github.com/percent4/tensorflow_serving_examples 。
感謝大家的閱讀，後續將介紹如何使用tensorflow/serving來部署BERT模型。