We studied the transferability of MTEB and publicly available ecommerce dataset benchmark performance to real-world retail search applications. The results in this white paper underscore the necessity for domain-specific evaluations and benchmarks.