前篇：EfficientNet v1筆記
個人總結
亮點1：改進了efficientnet模型
亮點2：提出了漸進式學習方法（個人覺得思路很棒，因為類似於人類的學習方式，隨著年齡增加，我們學習內容的難度也是由簡單到困難）
訓練前期，輸入較小尺寸的圖片，使用較弱的正則化策略（Dropout、RandAugment、Mixup）；
訓練後期，輸入較大尺寸的圖片，使用較強的正則化策略。
通過漸進式學習，訓練時間有很大的縮減，且模型性能有稍許提高：
EfficientNetV2: Smaller Models and Faster Training
谷歌 2021 CVPR
介紹
首先指出了EfficientNet v1存在的問題：
（1）訓練圖像的尺寸很大時，訓練速度非常慢
（2）在網絡淺層中使用DW卷積速度會非常慢
（3）同等放大每個stage是次優的
本文使用訓練感知神經結構搜索和縮放，共同優化訓練速度和參數效率。該模型從增加了新的操作（Fused-MBConv）的搜索空間中搜索。實驗表明，EffecientNet v2模型的訓練速度比最先進的模型快得多，而體積卻小了6.8倍。
本文還通過漸進學習加快訓練速度：在早期訓練階段，我們用小圖像尺寸和弱正則化(如dropout和data augmentation)訓練網絡，然後我們逐漸增加圖像尺寸和增強正則化。基於漸進式調整(Howard,2018)，我們的方法可以在不導致準確性下降的情況下加快訓練速度。
性能圖：
網絡框架（EfficientNetV2-S）：
附（EfficienctNetv1）：
與v1相比：
（1）混合使用了MBConv和Fused-MBConv（Fused-MBConv在實際代碼中無SE模塊）；
（2）使用了較小的expansion rate；
（3）kernel size也變小了；
（4）移除了v1中最後一個步距為1的stage，即v1中的stage8。
代碼實現
SE模塊：
class SqueezeExcite(nn.Module): def __init__(self, input_c: int, # MBblock input channel expand_c: int, # block expand channel（DW卷積的輸出channel） se_ratio: float = 0.25): super(SqueezeExcite, self).__init__() squeeze_c = int(input_c * se_ratio) self.conv_reduce = nn.Conv2d(expand_c, squeeze_c, 1) self.act1 = nn.SiLU() # Swish激活函數 self.conv_expand = nn.Conv2d(squeeze_c, expand_c, 1) self.act2 = nn.Sigmoid()
def forward(self, x: Tensor) -> Tensor: scale = x.mean((2, 3), keepdim=True) scale = self.conv_reduce(scale) scale = self.act1(scale) scale = self.conv_expand(scale) scale = self.act2(scale) return scale * xMBConv（有捷徑分支的時候才有Dropout層）：
dropout層：
def drop_path(x, drop_prob: float = 0., training: bool = False): """ Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). "Deep Networks with Stochastic Depth", https://arxiv.org/pdf/1603.09382.pdf
This function is taken from the rwightman. It can be seen here: https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/layers/drop.py#L140 """ if drop_prob == 0. or not training: return x keep_prob = 1 - drop_prob shape = (x.shape[0],) + (1,) * (x.ndim - 1) # work with diff dim tensors, not just 2D ConvNets random_tensor = keep_prob + torch.rand(shape, dtype=x.dtype, device=x.device) random_tensor.floor_() # binarize output = x.div(keep_prob) * random_tensor return output
class DropPath(nn.Module): """ Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). "Deep Networks with Stochastic Depth", https://arxiv.org/pdf/1603.09382.pdf """ def __init__(self, drop_prob=None): super(DropPath, self).__init__() self.drop_prob = drop_prob
def forward(self, x): return drop_path(x, self.drop_prob, self.training)MBConv：
class MBConv(nn.Module): def __init__(self, kernel_size: int, input_c: int, out_c: int, expand_ratio: int, stride: int, se_ratio: float, drop_rate: float, norm_layer: Callable[..., nn.Module]): super(MBConv, self).__init__()
if stride not in [1, 2]: raise ValueError("illegal stride value.")
self.has_shortcut = (stride == 1 and input_c == out_c)
activation_layer = nn.SiLU expanded_c = input_c * expand_ratio
# 在EfficientNetV2中，MBConv中不存在expansion=1的情況所以conv_pw肯定存在 assert expand_ratio != 1 # Point-wise expansion self.expand_conv = ConvBNAct(input_c, expanded_c, kernel_size=1, norm_layer=norm_layer, activation_layer=activation_layer)
# Depth-wise convolution self.dwconv = ConvBNAct(expanded_c, expanded_c, kernel_size=kernel_size, stride=stride, groups=expanded_c, norm_layer=norm_layer, activation_layer=activation_layer)
self.se = SqueezeExcite(input_c, expanded_c, se_ratio)
# Point-wise linear projection self.project_conv = ConvBNAct(expanded_c, out_planes=out_c, kernel_size=1, norm_layer=norm_layer, activation_layer=nn.Identity) # 注意這裡沒有激活函數，所有傳入Identity
self.out_channels = out_c
# 只有在使用shortcut連接時才使用dropout層 self.drop_rate = drop_rate if self.has_shortcut and drop_rate > 0: self.dropout = DropPath(drop_rate)
def forward(self, x: Tensor) -> Tensor: result = self.expand_conv(x) result = self.dwconv(result) result = self.se(result) result = self.project_conv(result)
if self.has_shortcut: if self.drop_rate > 0: result = self.dropout(result) result += x
return resultFused-MBConv：
class FusedMBConv(nn.Module): def __init__(self, kernel_size: int, input_c: int, out_c: int, expand_ratio: int, stride: int, se_ratio: float, drop_rate: float, norm_layer: Callable[..., nn.Module]): super(FusedMBConv, self).__init__()
assert stride in [1, 2] assert se_ratio == 0
self.has_shortcut = stride == 1 and input_c == out_c self.drop_rate = drop_rate
self.has_expansion = expand_ratio != 1
activation_layer = nn.SiLU # alias Swish expanded_c = input_c * expand_ratio
# 只有當expand ratio不等於1時才有expand conv if self.has_expansion: # Expansion convolution self.expand_conv = ConvBNAct(input_c, expanded_c, kernel_size=kernel_size, stride=stride, norm_layer=norm_layer, activation_layer=activation_layer)
self.project_conv = ConvBNAct(expanded_c, out_c, kernel_size=1, norm_layer=norm_layer, activation_layer=nn.Identity) # 注意沒有激活函數 else: # 當只有project_conv時的情況 self.project_conv = ConvBNAct(input_c, out_c, kernel_size=kernel_size, stride=stride, norm_layer=norm_layer, activation_layer=activation_layer) # 注意有激活函數
self.out_channels = out_c
# 只有在使用shortcut連接時才使用dropout層 self.drop_rate = drop_rate if self.has_shortcut and drop_rate > 0: self.dropout = DropPath(drop_rate)
def forward(self, x: Tensor) -> Tensor: if self.has_expansion: result = self.expand_conv(x) result = self.project_conv(result) else: result = self.project_conv(x)
if self.has_shortcut: if self.drop_rate > 0: result = self.dropout(result)
result += x
return result配置文件：
def efficientnetv2_s(num_classes: int = 1000): """ EfficientNetV2 https://arxiv.org/abs/2104.00298 """ # train_size: 300, eval_size: 384
# repeat, kernel, stride, expansion, in_c, out_c, operator, se_ratio model_config = [[2, 3, 1, 1, 24, 24, 0, 0], [4, 3, 2, 4, 24, 48, 0, 0], [4, 3, 2, 4, 48, 64, 0, 0], [6, 3, 2, 4, 64, 128, 1, 0.25], [9, 3, 1, 6, 128, 160, 1, 0.25], [15, 3, 2, 6, 160, 256, 1, 0.25]]
model = EfficientNetV2(model_cnf=model_config, num_classes=num_classes, dropout_rate=0.2) return model
def efficientnetv2_m(num_classes: int = 1000): """ EfficientNetV2 https://arxiv.org/abs/2104.00298 """ # train_size: 384, eval_size: 480
# repeat, kernel, stride, expansion, in_c, out_c, operator, se_ratio model_config = [[3, 3, 1, 1, 24, 24, 0, 0], [5, 3, 2, 4, 24, 48, 0, 0], [5, 3, 2, 4, 48, 80, 0, 0], [7, 3, 2, 4, 80, 160, 1, 0.25], [14, 3, 1, 6, 160, 176, 1, 0.25], [18, 3, 2, 6, 176, 304, 1, 0.25], [5, 3, 1, 6, 304, 512, 1, 0.25]]
model = EfficientNetV2(model_cnf=model_config, num_classes=num_classes, dropout_rate=0.3) return model
def efficientnetv2_l(num_classes: int = 1000): """ EfficientNetV2 https://arxiv.org/abs/2104.00298 """ # train_size: 384, eval_size: 480
# repeat, kernel, stride, expansion, in_c, out_c, operator, se_ratio model_config = [[4, 3, 1, 1, 32, 32, 0, 0], [7, 3, 2, 4, 32, 64, 0, 0], [7, 3, 2, 4, 64, 96, 0, 0], [10, 3, 2, 4, 96, 192, 1, 0.25], [19, 3, 1, 6, 192, 224, 1, 0.25], [25, 3, 2, 6, 224, 384, 1, 0.25], [7, 3, 1, 6, 384, 640, 1, 0.25]]
model = EfficientNetV2(model_cnf=model_config, num_classes=num_classes, dropout_rate=0.4) return model整體架構：
class EfficientNetV2(nn.Module): def __init__(self, model_cnf: list, num_classes: int = 1000, num_features: int = 1280, dropout_rate: float = 0.2, drop_connect_rate: float = 0.2): super(EfficientNetV2, self).__init__()
for cnf in model_cnf: assert len(cnf) == 8
norm_layer = partial(nn.BatchNorm2d, eps=1e-3, momentum=0.1)
stem_filter_num = model_cnf[0][4]
self.stem = ConvBNAct(3, stem_filter_num, kernel_size=3, stride=2, norm_layer=norm_layer) # 激活函數默認是SiLU
total_blocks = sum([i[0] for i in model_cnf]) block_id = 0 blocks = [] for cnf in model_cnf: repeats = cnf[0] op = FusedMBConv if cnf[-2] == 0 else MBConv for i in range(repeats): blocks.append(op(kernel_size=cnf[1], input_c=cnf[4] if i == 0 else cnf[5], out_c=cnf[5], expand_ratio=cnf[3], stride=cnf[2] if i == 0 else 1, se_ratio=cnf[-1], drop_rate=drop_connect_rate * block_id / total_blocks, norm_layer=norm_layer)) block_id += 1 self.blocks = nn.Sequential(*blocks)
head_input_c = model_cnf[-1][-3] head = OrderedDict()
head.update({"project_conv": ConvBNAct(head_input_c, num_features, kernel_size=1, norm_layer=norm_layer)}) # 激活函數默認是SiLU
head.update({"avgpool": nn.AdaptiveAvgPool2d(1)}) head.update({"flatten": nn.Flatten()})
if dropout_rate > 0: head.update({"dropout": nn.Dropout(p=dropout_rate, inplace=True)}) head.update({"classifier": nn.Linear(num_features, num_classes)})
self.head = nn.Sequential(head)
# initial weights for m in self.modules(): if isinstance(m, nn.Conv2d): nn.init.kaiming_normal_(m.weight, mode="fan_out") if m.bias is not None: nn.init.zeros_(m.bias) elif isinstance(m, nn.BatchNorm2d): nn.init.ones_(m.weight) nn.init.zeros_(m.bias) elif isinstance(m, nn.Linear): nn.init.normal_(m.weight, 0, 0.01) nn.init.zeros_(m.bias)
def forward(self, x: Tensor) -> Tensor: x = self.stem(x) x = self.blocks(x) x = self.head(x)
return x參考：
原論文
霹靂吧啦Wz（bilibili）