python .\train.py --dataset .\posts.npz --model_name 117M
2021-07-26 18:56:22.744751: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-07-26 18:56:25.665370: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2021-07-26 18:56:25.690352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1060 6GB computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 10 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 178.99GiB/s
2021-07-26 18:56:25.690809: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2021-07-26 18:56:25.702755: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2021-07-26 18:56:25.702942: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2021-07-26 18:56:25.714849: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2021-07-26 18:56:25.716629: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2021-07-26 18:56:25.721642: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2021-07-26 18:56:25.726209: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2021-07-26 18:56:25.727280: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2021-07-26 18:56:25.727816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-07-26 18:56:25.728573: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-07-26 18:56:25.729437: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1060 6GB computeCapability: 6.1
coreClock: 1.7085GHz coreCount: 10 deviceMemorySize: 6.00GiB deviceMemoryBandwidth: 178.99GiB/s
2021-07-26 18:56:25.729755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-07-26 18:56:26.368525: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-26 18:56:26.368696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
2021-07-26 18:56:26.370010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
2021-07-26 18:56:26.370859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4624 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
WARNING:tensorflow:From D:\Users\User\code\gpt-2\src\sample.py:60: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From D:\Users\User\code\gpt-2\.venv\lib\site-packages\tensorflow\python\util\dispatch.py:206: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.random.categorical` instead.
Using Adam optimizer
2021-07-26 18:56:38.717126: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 4.52G (4848943104 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-07-26 18:56:38.825003: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 4.06G (4364048896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-07-26 18:56:38.932709: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 3.66G (3927643904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-07-26 18:56:39.039233: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 3.29G (3534879488 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-07-26 18:56:39.147241: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 2.96G (3181391360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory