tensorflow訓(xùn)練中出現(xiàn)nan問(wèn)題的解決-創(chuàng)新互聯(lián)
深度學(xué)習(xí)中對(duì)于網(wǎng)絡(luò)的訓(xùn)練是參數(shù)更新的過(guò)程，需要注意一種情況就是輸入數(shù)據(jù)未做歸一化時(shí)，如果前向傳播結(jié)果已經(jīng)是[0,0,0,1,0,0,0,0]這種形式，而真實(shí)結(jié)果是[1,0,0,0,0,0,0,0,0]，此時(shí)由于得出的結(jié)論不懼有概率性，而是錯(cuò)誤的估計(jì)值，此時(shí)反向傳播會(huì)使得權(quán)重和偏置值變的無(wú)窮大，導(dǎo)致數(shù)據(jù)溢出，也就出現(xiàn)了nan的問(wèn)題。
解決辦法：
1、對(duì)輸入數(shù)據(jù)進(jìn)行歸一化處理，如將輸入的圖片數(shù)據(jù)除以255將其轉(zhuǎn)化成0-1之間的數(shù)據(jù);
2、對(duì)于層數(shù)較多的情況，各層都做batch_nomorlization;
3、對(duì)設(shè)置Weights權(quán)重使用tf.truncated_normal(0, 0.01, [3,3,1,64])生成，同時(shí)值的均值為0，方差要小一些;
4、激活函數(shù)可以使用tanh;
5、減小學(xué)習(xí)率lr。
實(shí)例：
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('data',one_hot = True)
def add_layer(input_data,in_size, out_size,activation_function=None):
Weights = tf.Variable(tf.random_normal([in_size,out_size]))
Biases = tf.Variable(tf.zeros([1, out_size])+0.1)
Wx_plus_b = tf.add(tf.matmul(input_data, Weights), Biases)
if activation_function==None:
outputs = Wx_plus_b
else:
outputs = activation_function(Wx_plus_b)
#return outputs#, Weights
return {'outdata':outputs, 'w':Weights}
def get_accuracy(t_y):
# global l1
# accu = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(l1['outdata'],1),tf.argmax(t_y,1)), dtype = tf.float32))
global prediction
accu = tf.reduce_mean(tf.cast(tf.equal(tf.argmax(prediction['outdata'],1),tf.argmax(t_y,1)), dtype = tf.float32))
return accu
X = tf.placeholder(tf.float32, [None, 784])
Y = tf.placeholder(tf.float32, [None, 10])
#l1 = add_layer(X, 784, 10, tf.nn.softmax)
#cross_entropy = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(l1['outdata']), reduction_indices= [1]))
#l1 = add_layer(X, 784, 1024, tf.nn.relu)
l1 = add_layer(X, 784, 1024, None)
prediction = add_layer(l1['outdata'], 1024, 10, tf.nn.softmax)
cross_entropy = tf.reduce_mean(-tf.reduce_sum(Y*tf.log(prediction['outdata']), reduction_indices= [1]))
optimizer = tf.train.GradientDescentOptimizer(0.000001)
train = optimizer.minimize(cross_entropy)
newW = tf.Variable(tf.random_normal([1024,10]))
newOut = tf.matmul(l1['outdata'],newW)
newSoftMax = tf.nn.softmax(newOut)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
#print(sess.run(l1_Weights))
for i in range(2):
X_train, y_train = mnist.train.next_batch(1)
X_train = X_train/255 #需要進(jìn)行歸一化處理
#print(sess.run(l1['w'],feed_dict={X:X_train}))
#print(sess.run(prediction['w'],feed_dict={X:X_train, Y:y_train}))
#print(sess.run(l1['outdata'],feed_dict={X:X_train, Y:y_train}).shape)
print(sess.run(prediction['outdata'],feed_dict={X:X_train, Y:y_train}))
print(sess.run(newOut, feed_dict={X:X_train}))
print(sess.run(newSoftMax, feed_dict={X:X_train}))
print(y_train)
#print(sess.run(l1['outdata'], feed_dict={X:X_train}))
sess.run(train, feed_dict={X:X_train, Y:y_train})
if i%100 == 0:
#print(sess.run(cross_entropy, feed_dict={X:X_train, Y:y_train}))
accuracy = get_accuracy(mnist.test.labels)
print(sess.run(accuracy,feed_dict={X:mnist.test.images}))
#if i%100==0:
#print(sess.run(prediction, feed_dict={X:X_train}))
#print(sess.run(cross_entropy, feed_dict={X:X_train,Y:y_train}))
另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn，海內(nèi)外云服務(wù)器15元起步，三天無(wú)理由+7*72小時(shí)售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國(guó)服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡(jiǎn)單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢(shì)，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場(chǎng)景需求。
文章名稱：tensorflow訓(xùn)練中出現(xiàn)nan問(wèn)題的解決-創(chuàng)新互聯(lián)
網(wǎng)址分享：http://m.jiaotiyi.com/article/hpdgd.html