Model-based genotype and ancestry estimation for potential hybrids with mixed-ploidy